Copula Ordinal Regression Framework for Joint 2 Estimation of Facial Action Unit Intensity

4 Abstract—Joint modeling of the intensity Q1 of multiple facial action units (AUs) from face images is challenging due to the large number 5 of AUs (30+) and their intensity levels (6). This is in part due to the lack of suitable models that can efficiently handle such a large 6 number of outputs/classes simultaneously, but also due to the lack of suitable data the models on. For this reason, majority of the 7 methods resort to independent classifiers for the AU intensity. This is suboptimal for at least two reasons: the facial appearance of 8 some AUs changes depending on the intensity of other AUs, and some AUs co-occur more often than others. To this end, we propose 9 the Copula regression approach for modeling multivariate ordinal variables. Our model accounts for ordinal structure in output 10 variables and their non-linear dependencies via copula functions modeled as cliques of a conditional random fields. The copula ordinal 11 regression model achieves the joint learning and inference of intensities of multiple AUs, while being computationally tractable. We 12 demonstrate the effectiveness of our approach on three challenging datasets of naturalistic facial expressions and we show that the 13 estimation of target AU intensities improves especially in the case of (a) noisy image features, (b) head-pose variations and (c) 14 imbalanced training data. Lastly, we show that the proposed approach consistently outperforms (i) independent modeling of AU 15 intensities and (ii) the state-of-the-art approach for the target task and (iii) deep convolutional neural networks.